{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "W9_Homework.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KKgzsidLGINz"
      },
      "source": [
        "# Week 9 Homework"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "d1GTaygz4aqK",
        "cellView": "form"
      },
      "source": [
        "#@markdown What is your Pennkey and pod? (text, not numbers, e.g. bfranklin)\n",
        "my_pennkey = '' #@param {type:\"string\"}\n",
        "my_pod = 'Select' #@param ['Select', 'euclidean-wombat', 'sublime-newt', 'buoyant-unicorn', 'lackadaisical-manatee','indelible-stingray','superfluous-lyrebird','discreet-reindeer','quizzical-goldfish','astute-jellyfish','ubiquitous-cheetah','nonchalant-crocodile','fashionable-lemur','spiffy-eagle','electric-emu','quotidian-lion']\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ajtPqks9C1Qp"
      },
      "source": [
        "## Section 1: Role of social media platforms in mitigating hate speech"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zzCVG642DFed"
      },
      "source": [
        "Social media platforms have instituted not just user policies for online conduct, but also technological mechanisms to identify and remove hate speech.  As an example, review Facebook’s discussion of how its employing AI to identify hate speech.  As you read this article, pay particular attention to the specific challenges they enumerate in identifying hate speech. \n",
        "\n",
        "\"AI advances to better detect hate speech\" \n",
        "\n",
        "https://ai.facebook.com/blog/ai-advances-to-better-detect-hate-speech/\n",
        "\n",
        "*Facebook, 2020* [10-minute read]\n",
        "\n",
        "The Facebook article emphasizes that the challenges to identifying hate speech are both social and technological.  The social considerations include the importance of cultural context as well as the nuances of languages. How does that resonate with the \"Excavating AI: The Politics of Images in Machine Learning Training Sets\" article you read in Week 6?  What do you think of Facebook’s approach? (200-300 words)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MOLhvJ8E4pYD",
        "cellView": "form"
      },
      "source": [
        "section1_ans = '' #@param {type:\"string\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fIgRA_GXRkLh"
      },
      "source": [
        "## Section 2: Broader societal considerations of addressing hate speech on social media\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0_J--GOXC4Bj"
      },
      "source": [
        "Next, read this Medium article about the unintended consequences of technological interventions to address hate speech on social media.\n",
        "\n",
        "\"How Automated Tools Discriminate Against Black Language\"    \n",
        "https://medium.com/@annawchung/how-automated-tools-discriminate-against-black-language-2ac8eab8d6db.  \n",
        "*Medium, 2019* [14-minute read]\n",
        "\n",
        "\"Challenges in Automated Debiasing for Toxic Language Detection\"    \n",
        "https://arxiv.org/pdf/2102.00086.pdf.  \n",
        "*Optional – this is much longer and more technical!!!*\n",
        "\n",
        "How does this example illustrate the complexity of identifying hate speech?  What solutions does the author suggest? Why is it important that AI practitioners have a robust understanding of the societal context and consequence of their work? (1-2 paragraphs)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "0HhP9MqW41tY"
      },
      "source": [
        "section2_ans = '' #@param {type:\"string\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d3A3u4yjC6B-"
      },
      "source": [
        "## Section 3: Ablation Study on LSTMs"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HujG0E6VUXzx"
      },
      "source": [
        "In the tutorials we learnt about the different gates in LSTMs. We also learnt about GRUs that have fewer components than LSTMs but work reasonably well. In this section of the homework you will conduct an ablation study for LSTMs. You will be working on text generation task. The model will be trained on H. G. Wells’ *The Time Machine*.\n",
        "\n",
        "In order to have a baseline, first intialize the weights and implement the gates of a LSTM. \n",
        "\n",
        "For the ablation study change different components listed below (these are suggestions, you are free to use your own ideas as well) and observe the effects on performance. \n",
        "- Initialization\n",
        "- Number of gates\n",
        "- Replacing sigmoid, tanh with other functions\n",
        "- Changing the inputs used for each recurrent unit\n",
        "\n",
        "You will make changes in the `get_lstm_params` and `lstm` functions. We have provided code for forward and backward propagation, training and a sample code to see the text generation results. \n",
        "(Please remember to to return tensors with the same shape as the LSTM you will implement to ensure end to end training.)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "C8F7j01s3eu_",
        "cellView": "form"
      },
      "source": [
        "#@title Imports\n",
        "!pip install d2l\n",
        "\n",
        "import torch\n",
        "from torch import nn\n",
        "from d2l import torch as d2l\n",
        "from torch.nn import functional as F\n",
        "import math"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MiJvAbGP3pup"
      },
      "source": [
        "def get_lstm_params(vocab_size, num_hiddens):\n",
        "    \n",
        "    num_inputs = num_outputs = vocab_size\n",
        "\n",
        "    ''' <TODO> '''\n",
        "    ...\n",
        "    ''' </TODO> '''\n",
        "    \n",
        "    # List all the parameters and attach gradients\n",
        "    params = [\n",
        "        W_xi, W_hi, b_i, W_xf, W_hf, b_f, W_xo, W_ho, b_o, W_xc, W_hc, b_c,\n",
        "        W_hq, b_q]\n",
        "    for param in params:\n",
        "        param.requires_grad_(True)\n",
        "    return params"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5pcMcEav3t24"
      },
      "source": [
        "def lstm(inputs, state, params):\n",
        "    # Get the model params\n",
        "    [W_xi, W_hi, b_i, W_xf, W_hf, b_f, W_xo, W_ho, b_o, W_xc, W_hc, b_c,\n",
        "        W_hq, b_q] = params\n",
        "    \n",
        "    # Get the hidden and cell state\n",
        "    (H, C) = state\n",
        "\n",
        "    # Store outputs for each input from a batch \n",
        "    outputs = []\n",
        "\n",
        "    # For each of the inputs in a batch, perform the LSTM operations\n",
        "    for X in inputs:\n",
        "        \n",
        "        ''' <TODO> '''\n",
        "        ...\n",
        "        ''' <TODO> '''\n",
        "\n",
        "    return torch.cat(outputs, dim=0), (H, C)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8PzTeciC3-e9"
      },
      "source": [
        "# Build the LSTM model from scratch\n",
        "class LSTMScratch:  \n",
        "    \n",
        "    \"\"\"An LSTM Model implemented from scratch.\"\"\"\n",
        "    \n",
        "    def __init__(self, vocab_size, num_hiddens, get_params, forward_fn):\n",
        "\n",
        "        self.vocab_size, self.num_hiddens, self.forward_fn = vocab_size, num_hiddens, forward_fn\n",
        "        # Function to initialize all the parameters in the LSTM (defined by you!)\n",
        "        self.params = get_params(vocab_size, num_hiddens)\n",
        "\n",
        "    def __call__(self, X, state):\n",
        "        # Calls the forward function \n",
        "        \n",
        "        # Convert X to a one-hot vector for each character instead of an Embedding\n",
        "        X = F.one_hot(X.T, self.vocab_size).type(torch.float32)\n",
        "        \n",
        "        # Function to pass inputs to the model (defined by you!)\n",
        "        return self.forward_fn(X, state, self.params)\n",
        "\n",
        "    def begin_state(self, batch_size):\n",
        "        # Initialize the hidden state and cell state with zeros\n",
        "        return (torch.zeros((batch_size, num_hiddens), device=device),\n",
        "            torch.zeros((batch_size, num_hiddens), device=device))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PYx7nc3u4XJz",
        "cellView": "form"
      },
      "source": [
        "#@title Training Functions (Run Me!)\n",
        "# Clip the gradients\n",
        "def grad_clipping(net, theta):\n",
        "    \"\"\"Clip the gradient.\"\"\"\n",
        "    params = net.params\n",
        "    norm = torch.sqrt(sum(torch.sum((p.grad**2)) for p in params))\n",
        "    if norm > theta:\n",
        "        for param in params:\n",
        "            param.grad[:] *= theta / norm\n",
        "\n",
        "# Optimization Minibatch SGD\n",
        "def sgd(params, lr, batch_size):\n",
        "    \"\"\"Minibatch stochastic gradient descent.\"\"\"\n",
        "    with torch.no_grad():\n",
        "        for param in params:\n",
        "            param -= lr * param.grad / batch_size\n",
        "            param.grad.zero_()\n",
        "\n",
        "# Training in each epoch\n",
        "def train_epoch(net, train_iter, loss, optimizer):\n",
        "    \"\"\"Train a net within one epoch (defined in Chapter 8).\"\"\"\n",
        "\n",
        "    state, timer = None, d2l.Timer()\n",
        "    metric = d2l.Accumulator(2)  # Sum of training loss, no. of tokens\n",
        "    \n",
        "    # For every barch in dataloader\n",
        "    for X, Y in train_iter:\n",
        "        if state is None:\n",
        "            # Initialize `state` when either it is the first iteration or\n",
        "            # using random sampling\n",
        "            state = net.begin_state(batch_size=X.shape[0])\n",
        "        else:\n",
        "            # `state` is a tuple of tensors for our custom scratch implementation\n",
        "            for s in state:\n",
        "                s.detach_()\n",
        "\n",
        "        # Pass input and state to the model\n",
        "        y = Y.T.reshape(-1)\n",
        "        X, y = X.to(device), y.to(device)\n",
        "        y_hat, state = net(X, state)\n",
        "\n",
        "        # Compute loss, backpropagate and clip the gradient\n",
        "        l = loss(y_hat, y.long()).mean()\n",
        "        l.backward()\n",
        "        grad_clipping(net, 1)\n",
        "\n",
        "        # Step the optimizer\n",
        "        optimizer(batch_size=1)\n",
        "\n",
        "        # Loss metric append    \n",
        "        metric.add(l * y.numel(), y.numel())\n",
        "    return math.exp(metric[0] / metric[1]), metric[1] / timer.stop()\n",
        "\n",
        "# Predict function\n",
        "def predict(prefix, num_preds, net, vocab):  \n",
        "    \"\"\"Generate new characters following the `prefix`.\"\"\"    \n",
        "    # Initialize and tensorize input\n",
        "    state = net.begin_state(batch_size=1)\n",
        "    outputs = [vocab[prefix[0]]]\n",
        "    get_input = lambda: torch.tensor([outputs[-1]], device=device).reshape(\n",
        "        (1, 1))\n",
        "    \n",
        "    for y in prefix[1:]:  # Warm-up period\n",
        "        _, state = net(get_input(), state)\n",
        "        outputs.append(vocab[y])\n",
        "    \n",
        "    for _ in range(num_preds):  # Predict `num_preds` steps\n",
        "        y, state = net(get_input(), state)\n",
        "        outputs.append(int(y.argmax(dim=1).reshape(1)))\n",
        "    \n",
        "    return ''.join([vocab.idx_to_token[i] for i in outputs])\n",
        "\n",
        "# Train function\n",
        "def train(net, train_iter, vocab, lr, num_epochs):\n",
        "    \"\"\"Train a model.\"\"\"\n",
        "\n",
        "    # Loss function and loss visualization tool \n",
        "    loss = nn.CrossEntropyLoss()\n",
        "    animator = d2l.Animator(xlabel='epoch', ylabel='perplexity', legend=['train'], xlim=[10, num_epochs])\n",
        "    \n",
        "    # Optimizer\n",
        "    optimizer = lambda batch_size: sgd(net.params, lr, batch_size)\n",
        "\n",
        "    # Train and predict over each epoch\n",
        "    for epoch in range(num_epochs):\n",
        "        ppl, speed = train_epoch(net, train_iter, loss, optimizer)\n",
        "        \n",
        "        # Plot every 10 epochs\n",
        "        if (epoch + 1) % 10 == 0:\n",
        "            print(predict('time traveller', 50, net, vocab))\n",
        "            animator.add(epoch + 1, [ppl])\n",
        "    \n",
        "    # Final predictions\n",
        "    print(f'perplexity {ppl:.1f}, {speed:.1f} tokens/sec on {str(device)}')\n",
        "    print(predict('time traveller', 50, net, vocab))\n",
        "    print(predict('traveller', 50, net, vocab))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MegqMA8pGTXe"
      },
      "source": [
        "batch_size, num_steps = 32, 50\n",
        "\n",
        "train_iter, vocab = d2l.load_data_time_machine(batch_size, num_steps)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LcOy5DAR3wLD"
      },
      "source": [
        "vocab_size, num_hiddens, device = len(vocab), 256, torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
        "num_epochs, lr = 500, 1\n",
        "\n",
        "model = LSTMScratch(len(vocab), num_hiddens, get_lstm_params, lstm)\n",
        "train(model, train_iter, vocab, lr, num_epochs)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fykfRPEwHiai"
      },
      "source": [
        "# Add your initial primer strings to this list for testing\n",
        "sentences = ['time traveller',\n",
        "             'traveller',\n",
        "             'machine'\n",
        "\n",
        "]\n",
        "\n",
        "for sentence in sentences:\n",
        "    print(predict(sentence, 50, model, vocab))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "APi-IM5EVh7K"
      },
      "source": [
        "Summarize the results of your ablation study. What changes did you make? How was the performance affacted? Provide reasoning for your observations. (200-400 words)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "5CmtmN-j5Pk9"
      },
      "source": [
        "section3_ans = '' #@param {type:\"string\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "le-1JMKWRVxy"
      },
      "source": [
        "## Section 4: Know-Your-Pod Better"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_u3z3BIcRZkG"
      },
      "source": [
        "Discuss with two other members of your pod. What is their favorite movie scene (not just the whole movie, but describe the scene!) of all time? Describe the scene and why they like it so much. (~100 words each)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "AbIh2_ca44DK"
      },
      "source": [
        "section4_ans = '' #@param {type:\"string\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gbgTqgB5rH_W"
      },
      "source": [
        "---\n",
        "# Submission\n",
        "Once you're done, click on 'Share' and add the link to the box below.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oeC-YYf85J2P",
        "cellView": "form"
      },
      "source": [
        "import time\n",
        "import numpy as np\n",
        "import urllib.parse\n",
        "from IPython.display import IFrame\n",
        "\n",
        "t7 = time.time()\n",
        "\n",
        "#@markdown #Run Cell to Show Airtable Form\n",
        "#@markdown ##**Confirm your answers and then click \"Submit\"**\n",
        "\n",
        "\n",
        "def prefill_form(src, fields: dict):\n",
        "  '''\n",
        "  src: the original src url to embed the form\n",
        "  fields: a dictionary of field:value pairs,\n",
        "  e.g. {\"pennkey\": my_pennkey, \"location\": my_location}\n",
        "  '''\n",
        "  prefill_fields = {}\n",
        "  for key in fields:\n",
        "      new_key = 'prefill_' + key\n",
        "      prefill_fields[new_key] = fields[key]\n",
        "  prefills = urllib.parse.urlencode(prefill_fields)\n",
        "  src = src + prefills\n",
        "  return src\n",
        "\n",
        "\n",
        "\n",
        "#autofill fields if they are not present\n",
        "#a missing pennkey and pod will result in an Airtable warning\n",
        "#which is easily fixed user-side.\n",
        "try: my_pennkey;\n",
        "except NameError: my_pennkey = \"\"\n",
        "\n",
        "try: my_pod;\n",
        "except NameError: my_pod = \"Select\"\n",
        "\n",
        "try: section1_ans;\n",
        "except NameError: section1_ans = \"\"\n",
        "\n",
        "try: section2_ans;\n",
        "except NameError: section2_ans = \"\"\n",
        "\n",
        "try: section3_ans;\n",
        "except NameError: section3_ans = \"\"\n",
        "\n",
        "try: section4_ans;\n",
        "except NameError: section4_ans = \"\"\n",
        "\n",
        "\n",
        "fields = {\"pennkey\": my_pennkey,\n",
        "          \"pod\": my_pod,\n",
        "          \"section1_ans\": section1_ans,\n",
        "          \"section2_ans\": section2_ans,\n",
        "          \"section3_ans\": section3_ans,\n",
        "          \"section4_ans\": section4_ans}\n",
        "\n",
        "src = \"https://airtable.com/shrZdjQjhvUSHVgW5?\"\n",
        "\n",
        "\n",
        "#now instead of the original source url, we do: src = prefill_form(src, fields)\n",
        "display.display(IFrame(src = prefill_form(src, fields), width = 800, height = 400))"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}